

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>清理队列 &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/css/custom.css" type="text/css" />

  
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/jquery.js"></script>
        <script src="../../_static/_sphinx_javascript_frameworks_compat.js"></script>
        <script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/doctools.js"></script>
        <script src="../../_static/sphinx_highlight.js"></script>
    
    <script type="text/javascript" src="../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../genindex/" />
    <link rel="search" title="Search" href="../../search/" />
    <link rel="next" title="客户端配置" href="../client-config-ref/" />
    <link rel="prev" title="CephFS 快照镜像" href="../cephfs-mirroring/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../../" class="icon icon-home" aria-label="Home"></a></li>
          <li class="breadcrumb-item"><a href="../">Ceph 文件系统</a></li>
      <li class="breadcrumb-item active">清理队列</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../../_sources/cephfs/purge-queue.rst.txt" rel="nofollow"> View page source</a>
      </li>
  </ul>
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../" class="icon icon-home"> Ceph
          

          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../start/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephadm/">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../">Ceph 文件系统</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../#cephfs">CephFS 入门</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../#id4">管理</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../createfs/"> 创建 CephFS 文件系统</a></li>
<li class="toctree-l3"><a class="reference internal" href="../administration/"> 管理命令</a></li>
<li class="toctree-l3"><a class="reference internal" href="../multifs/"> 创建多个文件系统</a></li>
<li class="toctree-l3"><a class="reference internal" href="../add-remove-mds/"> 配备、增加、删除 MDS</a></li>
<li class="toctree-l3"><a class="reference internal" href="../standby/">术语</a></li>
<li class="toctree-l3"><a class="reference internal" href="../standby/#mds">MDS 守护进程的引用</a></li>
<li class="toctree-l3"><a class="reference internal" href="../standby/#id3">故障切换的管理</a></li>
<li class="toctree-l3"><a class="reference internal" href="../standby/#standby-replay">热备（ standby-replay ）的配置</a></li>
<li class="toctree-l3"><a class="reference internal" href="../standby/#mds-join-fs">配置 MDS 与文件系统的亲和性</a></li>
<li class="toctree-l3"><a class="reference internal" href="../cache-configuration/"> MDS 缓存配置</a></li>
<li class="toctree-l3"><a class="reference internal" href="../mds-config-ref/"> MDS 配置选项</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../man/8/ceph-mds/"> ceph-mds 手册页</a></li>
<li class="toctree-l3"><a class="reference internal" href="../nfs/"> 通过 NFS 导出</a></li>
<li class="toctree-l3"><a class="reference internal" href="../app-best-practices/"> 应用最佳实践</a></li>
<li class="toctree-l3"><a class="reference internal" href="../fs-volumes/"> FS 卷和子卷</a></li>
<li class="toctree-l3"><a class="reference internal" href="../quota/"> CephFS 配额管理</a></li>
<li class="toctree-l3"><a class="reference internal" href="../health-messages/"> 健康消息</a></li>
<li class="toctree-l3"><a class="reference internal" href="../upgrading/">升级 MDS 集群</a></li>
<li class="toctree-l3"><a class="reference internal" href="../upgrading/#firefly-jewel">升级比 Firefly 老的文件系统，需过 Jewel 这个槛</a></li>
<li class="toctree-l3"><a class="reference internal" href="../cephfs-top/"> CephFS Top 工具</a></li>
<li class="toctree-l3"><a class="reference internal" href="../snap-schedule/"> 定时快照</a></li>
<li class="toctree-l3"><a class="reference internal" href="../cephfs-mirroring/"> CephFS 快照镜像</a></li>
<li class="toctree-l3 current"><a class="current reference internal" href="#"> 清理队列</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#id2">删除过程</a></li>
<li class="toctree-l4"><a class="reference internal" href="#perf">检查清理队列的 perf 计数器</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../#id5">挂载 CephFS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../#id6">CephFS 内幕</a></li>
<li class="toctree-l2"><a class="reference internal" href="../#id7">故障排除和灾难恢复</a></li>
<li class="toctree-l2"><a class="reference internal" href="../#id9">更多细节</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../monitoring/">监控概览</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dev/developer_guide/">开发者指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dev/internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../hardware-monitoring/">硬件监控</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <section id="id1">
<h1>清理队列<a class="headerlink" href="#id1" title="Permalink to this heading"></a></h1>
<p>MDS 维护着一个名为“清理队列（ Purge Queue ）”的数据结构，
负责管理和执行文件的并行删除。每个 MDS rank 都有一个清理队列。
清理队列由清理条目组成，其中包含来自 inode 的标称信息，
如大小和布局（即其他所有不需要的元数据信息都会忽略，
使其独立于所有元数据结构）。</p>
<section id="id2">
<h2>删除过程<a class="headerlink" href="#id2" title="Permalink to this heading"></a></h2>
<p>当一个客户端请求删除一个目录时（假设用 <code class="docutils literal notranslate"><span class="pre">rm</span> <span class="pre">-rf</span></code> ）：</p>
<ul class="simple">
<li><p>MDS 把 pq （清理队列， purge queue ）日志里的那些文件和子目录
（清理条目， purge items ）加进清理队列。</p></li>
<li><p>在后台以微小、且可管理的小块处理 inode 的删除。</p></li>
<li><p>MDS 发指令让底层的 OSD 们清理掉数据存储池里的相关对象。</p></li>
<li><p>更新日志</p></li>
</ul>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>如果用户删除文件的速度超过了清理队列的处理能力，
那么随着时间的推移，数据存储池的使用量可能会增加。
在极端情况下，清理队列积压得非常巨大，
以至于减缓了容量回收，此时，
CephFS 的 linux <code class="docutils literal notranslate"><span class="pre">du</span></code> 命令报告的数据可能与 CephFS 数据存储池不一致。</p>
</div>
<p>有几个可调节的配置， MDS 内部用它们节制清理队列的处理：</p>
<dl class="std confval">
<dt class="sig sig-object std" id="confval-filer_max_purge_ops">
<span class="sig-name descname"><span class="pre">filer_max_purge_ops</span></span><a class="headerlink" href="#confval-filer_max_purge_ops" title="Permalink to this definition"></a></dt>
<dd><blockquote>
<div><p>清理条带化存储空间（如 MDS 日志）的最大进行时（ in-flight ）操作数量。</p>
<dl class="field-list simple">
<dt class="field-odd">type<span class="colon">:</span></dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">uint</span></code></p>
</dd>
<dt class="field-even">default<span class="colon">:</span></dt>
<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">10</span></code></p>
</dd>
</dl>
</div></blockquote>
</dd></dl>

<dl class="std confval">
<dt class="sig sig-object std" id="confval-mds_max_purge_files">
<span class="sig-name descname"><span class="pre">mds_max_purge_files</span></span><a class="headerlink" href="#confval-mds_max_purge_files" title="Permalink to this definition"></a></dt>
<dd><blockquote>
<div><p>并行清理已删除文件的最大数量</p>
<dl class="field-list simple">
<dt class="field-odd">type<span class="colon">:</span></dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">uint</span></code></p>
</dd>
<dt class="field-even">default<span class="colon">:</span></dt>
<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">64</span></code></p>
</dd>
</dl>
</div></blockquote>
</dd></dl>

<dl class="std confval">
<dt class="sig sig-object std" id="confval-mds_max_purge_ops">
<span class="sig-name descname"><span class="pre">mds_max_purge_ops</span></span><a class="headerlink" href="#confval-mds_max_purge_ops" title="Permalink to this definition"></a></dt>
<dd><blockquote>
<div><p>并行执行清理操作的最大数量</p>
<dl class="field-list simple">
<dt class="field-odd">type<span class="colon">:</span></dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">uint</span></code></p>
</dd>
<dt class="field-even">default<span class="colon">:</span></dt>
<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">8Ki</span></code></p>
</dd>
</dl>
</div></blockquote>
</dd></dl>

<dl class="std confval">
<dt class="sig sig-object std" id="confval-mds_max_purge_ops_per_pg">
<span class="sig-name descname"><span class="pre">mds_max_purge_ops_per_pg</span></span><a class="headerlink" href="#confval-mds_max_purge_ops_per_pg" title="Permalink to this definition"></a></dt>
<dd><blockquote>
<div><p>每个 PG 执行并行清理操作的最大数量</p>
<dl class="field-list simple">
<dt class="field-odd">type<span class="colon">:</span></dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">float</span></code></p>
</dd>
<dt class="field-even">default<span class="colon">:</span></dt>
<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">0.5</span></code></p>
</dd>
</dl>
</div></blockquote>
</dd></dl>

<p>一般来说，默认值足以满足大多数集群的需要。不过，
如果集群非常庞大，需要 <code class="docutils literal notranslate"><span class="pre">pq_item_in_journal</span></code>
（待删除内容计数器）达到很大的数值，
那么可以试着把此配置调整为默认值的 4-5 倍，
进一步的增量取决于更大的需求。</p>
<p>从最微不足道的 <code class="docutils literal notranslate"><span class="pre">filer_max_purge_ops</span></code> 配置开始，
这应该有助于更快地回收空间：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph config set mds filer_max_purge_ops 40
</pre></div>
</div>
<p>对大多数集群来说，增加 <code class="docutils literal notranslate"><span class="pre">filer_max_purge_ops</span></code> 应该就够了，
但如果不行，那就继续调整其他配置选项：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph config set mds mds_max_purge_files 256
$ ceph config set mds mds_max_purge_ops 32768
$ ceph config set mds mds_max_purge_ops_per_pg 2
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>设置这些值不会立即破坏任何东西，
因为它们只是控制着我们能向底层 RADOS 集群发出多少删除操作，
如果设置的值高得惊人，
可能会牺牲一些集群性能。</p>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>就作业限制而言，与正在进行的工作相比，
清理队列并不是一个可以自动调整的系统。所以建议，在调整配置时，
应该根据集群规模和载荷做明智的决定。</p>
</div>
</section>
<section id="perf">
<h2>检查清理队列的 perf 计数器<a class="headerlink" href="#perf" title="Permalink to this heading"></a></h2>
<p>分析 MDS 的 perf dump 时，清理队列的统计信息大致如下：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="s2">&quot;purge_queue&quot;</span><span class="p">:</span> <span class="p">{</span>
    <span class="s2">&quot;pq_executing_ops&quot;</span><span class="p">:</span> <span class="mi">56655</span><span class="p">,</span>
    <span class="s2">&quot;pq_executing_ops_high_water&quot;</span><span class="p">:</span> <span class="mi">65350</span><span class="p">,</span>
    <span class="s2">&quot;pq_executing&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
    <span class="s2">&quot;pq_executing_high_water&quot;</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span>
    <span class="s2">&quot;pq_executed&quot;</span><span class="p">:</span> <span class="mi">25</span><span class="p">,</span>
    <span class="s2">&quot;pq_item_in_journal&quot;</span><span class="p">:</span> <span class="mi">6567004</span>
<span class="p">}</span>
</pre></div>
</div>
<p>我们了解一下每一项的含义：</p>
<table class="docutils align-default">
<colgroup>
<col style="width: 50.0%" />
<col style="width: 50.0%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>名字</p></th>
<th class="head"><p>描述</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p>pq_executing_ops</p></td>
<td><p>正在进行的清理队列操作数</p></td>
</tr>
<tr class="row-odd"><td><p>pq_executing_ops_high_water</p></td>
<td><p>记录到的正在执行的清理操作最大数量</p></td>
</tr>
<tr class="row-even"><td><p>pq_executing</p></td>
<td><p>正在删除的清理队列文件</p></td>
</tr>
<tr class="row-odd"><td><p>pq_executing_high_water</p></td>
<td><p>正在执行的文件清理最大数量</p></td>
</tr>
<tr class="row-even"><td><p>pq_executed</p></td>
<td><p>已删除的清理队列文件</p></td>
</tr>
<tr class="row-odd"><td><p>pq_item_in_journal</p></td>
<td><p>日志里残留的清理条目（文件）</p></td>
</tr>
</tbody>
</table>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p><code class="docutils literal notranslate"><span class="pre">pq_executing</span></code> 和 <code class="docutils literal notranslate"><span class="pre">pq_executing_ops</span></code> 看起来很相似，
但两者之间有细微差别。 <code class="docutils literal notranslate"><span class="pre">pq_executing</span></code> 追踪的是清理队列中的文件数量，
而 <code class="docutils literal notranslate"><span class="pre">pq_executing_ops</span></code> 统计的是清理队列中所有文件的 RADOS 对象数量。</p>
</div>
</section>
</section>



<div id="support-the-ceph-foundation" class="admonition note">
  <p class="first admonition-title">Brought to you by the Ceph Foundation</p>
  <p class="last">The Ceph Documentation is a community resource funded and hosted by the non-profit <a href="https://ceph.io/en/foundation/">Ceph Foundation</a>. If you would like to support this and our other efforts, please consider <a href="https://ceph.io/en/foundation/join/">joining now</a>.</p>
</div>


           </div>
           
          </div>
          <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
        <a href="../cephfs-mirroring/" class="btn btn-neutral float-left" title="CephFS 快照镜像" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="../client-config-ref/" class="btn btn-neutral float-right" title="客户端配置" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>&#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).</p>
  </div>

   

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>